Biostat/Biomath M257
This session introduces GPU computing in Julia.
GPUs are ubiquitous in modern computers. Following are GPUs today’s typical computer systems.
| NVIDIA GPUs | Tesla K80 | GTX 1080 | GT 650M |
|---|---|---|---|
 |
 |
 |
|
| Computers | servers, cluster | desktop | laptop |
 |
 |
 |
|
| Main usage | scientific computing | daily work, gaming | daily work |
| Memory | 24 GB | 8 GB | 1GB |
| Memory bandwidth | 480 GB/sec | 320 GB/sec | 80GB/sec |
| Number of cores | 4992 | 2560 | 384 |
| Processor clock | 562 MHz | 1.6 GHz | 0.9GHz |
| Peak DP performance | 2.91 TFLOPS | 257 GFLOPS | |
| Peak SP performance | 8.73 TFLOPS | 8228 GFLOPS | 691Gflops |
GPU architecture vs CPU architecture.
* GPUs contain 100s of processing cores on a single card; several cards can fit in a desktop PC
* Each core carries out the same operations in parallel on different input data – single program, multiple data (SPMD) paradigm
* Extremely high arithmetic intensity if one can transfer the data onto and results off of the processors quickly
 |
 |
|---|---|
 |
 |
GPU support by Julia is under active development. Check JuliaGPU for currently available packages.
There are multiple paradigms to program GPU in Julia, depending on the specific hardware.
CUDA is an ecosystem exclusively for Nvidia GPUs. There are extensive CUDA libraries for scientific computing: CuBLAS, CuRAND, CuSparse, CuSolve, CuDNN, …
The CUDA.jl package allows defining arrays on Nvidia GPUs and overloads many common operations.
The AMDGPU.jl package allows defining arrays on AMD GPUs and overloads many common operations.
The Metal.jl package allows defining arrays on Apple Silicon and overloads many common operations.
The oneAPI.jl package allows defining arrays on Intel GPUs and overloads many common operations.
I’ll illustrate using Metal.jl on my MacBook Pro running MacOS Ventura 13.2.1. It has Apple M2 chip with ??? GPU cores.
versioninfo()Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
OS: macOS (arm64-apple-darwin21.5.0)
CPU: 12 × Apple M2 Max
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
Threads: 1 on 8 virtual cores
Environment:
JULIA_EDITOR = codeLoad packages:
using Pkg
Pkg.activate(pwd())
Pkg.instantiate()
Pkg.status() Activating project at `~/Documents/github.com/ucla-biostat-257/2023spring/slides/09-juliagpu`Status `~/Documents/github.com/ucla-biostat-257/2023spring/slides/09-juliagpu/Project.toml`
[6e4b80f9] BenchmarkTools v1.3.2
[dde4c033] Metal v0.2.0
[37e2e46d] LinearAlgebrausing Metal
Metal.versioninfo()macOS 13.2.1, Darwin 21.5.0
Toolchain:
- Julia: 1.8.5
- LLVM: 13.0.1
1 device:
- Apple M2 Max (64.000 KiB allocated)# generate data on CPU
x = rand(Float32, 3, 3)
# transfer data form CPU to GPU
xd = MtlArray(x)3×3 MtlMatrix{Float32}:
0.776997 0.757277 0.576683
0.10972 0.626119 0.202182
0.628542 0.379558 0.0116938# generate array on GPU directly
# yd = Metal.ones(3, 3)
yd = MtlArray(ones(Float32, 3, 3))3×3 MtlMatrix{Float32}:
1.0 1.0 1.0
1.0 1.0 1.0
1.0 1.0 1.0# collect data from GPU to CPU
x = collect(xd)3×3 Matrix{Float32}:
0.776997 0.757277 0.576683
0.10972 0.626119 0.202182
0.628542 0.379558 0.0116938using BenchmarkTools, LinearAlgebra
n = 1024
# on CPU
x = rand(Float32, n, n)
y = rand(Float32, n, n)
z = zeros(Float32, n, n)
# on GPU
xd = MtlArray(x)
yd = MtlArray(y)
zd = MtlArray(z)
# SP matrix multiplication on GPU
@benchmark mul!($zd, $xd, $yd)BenchmarkTools.Trial: 10000 samples with 1 evaluation. Range (min … max): 9.541 μs … 725.333 μs ┊ GC (min … max): 0.00% … 0.00% Time (median): 246.042 μs ┊ GC (median): 0.00% Time (mean ± σ): 244.825 μs ± 26.842 μs ┊ GC (mean ± σ): 0.00% ± 0.00% ▁ ▂▃▅▇▇█▇▆▄▂ ▂ █▆▄▁▁▃▁▁▁▁▁▁▁▁▃▃▁▁▁▁▄▁▁▃▅▄▁▅▄▇▆▇▇▇▅▆▇▇▇▇████████████▇█▇▇▇▆▇▇▅ █ 9.54 μs Histogram: log(frequency) by time 321 μs < Memory estimate: 240 bytes, allocs estimate: 10.
# SP matrix multiplication on CPU
@benchmark mul!($z, $x, $y)BenchmarkTools.Trial: 1682 samples with 1 evaluation. Range (min … max): 2.876 ms … 5.982 ms ┊ GC (min … max): 0.00% … 0.00% Time (median): 2.971 ms ┊ GC (median): 0.00% Time (mean ± σ): 2.973 ms ± 89.342 μs ┊ GC (mean ± σ): 0.00% ± 0.00% ▁ ▁▁▃▆▇▇██▆▄▂ ▁ ██▆▅▄▅▅▅▄▆▅▆▅▇█▇█▆▇▅▇█████████████▇▇▅▅▅▆▅▄▁▄▄▅▄▁▁▄▅▄▄▅▄▄▄▅ █ 2.88 ms Histogram: log(frequency) by time 3.07 ms < Memory estimate: 0 bytes, allocs estimate: 0.
We see ~10x speedup in this matrix multiplication example.
# cholesky on Gram matrix
# This one doesn't seem to work on Apple M2 chip yet
xtxd = xd'xd + I
@benchmark cholesky($(Symmetric(xtxd)))LoadError: UndefVarError: N not definedxtx = collect(xtxd)
@benchmark cholesky($(Symmetric(xtx)))GPU speedup of Cholesky on this example is moderate.
# elementwise function on GPU arrays
fill!(yd, 1)
@benchmark $zd .= log.($yd .+ sin.($xd))BenchmarkTools.Trial: 10000 samples with 1 evaluation. Range (min … max): 113.500 μs … 18.587 ms ┊ GC (min … max): 0.00% … 87.36% Time (median): 135.750 μs ┊ GC (median): 0.00% Time (mean ± σ): 138.730 μs ± 184.695 μs ┊ GC (mean ± σ): 1.17% ± 0.87% ▁▂▄▅▅▅▆▇█▇█▇▇▇▇▅▄▄▃▂▂▁ ▂▁▁▁▂▂▂▂▂▂▂▃▄▄▆▇████████████████████████▇▆▆▆▅▅▄▄▄▄▃▃▃▃▂▃▃▃▂▂▂ ▅ 114 μs Histogram: frequency by time 162 μs < Memory estimate: 8.23 KiB, allocs estimate: 245.
# elementwise function on CPU arrays
x, y, z = collect(xd), collect(yd), collect(zd)
@benchmark $z .= log.($y .+ sin.($x))BenchmarkTools.Trial: 564 samples with 1 evaluation. Range (min … max): 8.494 ms … 9.433 ms ┊ GC (min … max): 0.00% … 0.00% Time (median): 8.866 ms ┊ GC (median): 0.00% Time (mean ± σ): 8.874 ms ± 96.558 μs ┊ GC (mean ± σ): 0.00% ± 0.00% ▄▇▆▅▁ █▇▃ ▂▂▂▁▁▁▁▂▂▁▂▂▁▁▂▃▂▁▃▅▄▅▆██████████▇▇▇▇▅▄▃▄▂▃▃▃▃▁▃▃▂▁▁▃▁▂▁▁▂ ▃ 8.49 ms Histogram: frequency by time 9.24 ms < Memory estimate: 0 bytes, allocs estimate: 0.
GPU brings great speedup (>500x) to the massive evaluation of elementary math functions.